Searching for user interface objects

ABSTRACT

A computer-implemented method includes searching resources related to an application in response to receiving a query, identifying an operation in the application that satisfies the query, and exposing an object that provides access to the identified operation, within the application.

This application is a continuation of U.S. application Ser. No.13/204,098, entitled “SEARCHING FOR USER INTERFACE OBJECTS,” filed onAug. 5, 2011, which is a continuation of U.S. application Ser. No.12/813,751, entitled “SEARCHING FOR USER INTERFACE OBJECTS,” filed onJun. 11, 2010, now U.S. Pat. No. 7,996,420, which is a continuation ofU.S. application Ser. No. 11/857,347, entitled “SEARCHING FOR USERINTERFACE OBJECTS,” filed Sep. 18, 2007, now U.S. Pat. No. 7,739,272,each assigned to the assignee of the present application. The subjectmatter of each of the above-identified commonly owned applications isincorporated herein by reference.

TECHNICAL FIELD

This disclosure relates to information retrieval.

BACKGROUND

Software applications installed on systems, e.g., computer systems,enable users to perform a variety of operations ranging from simple textentry to complicated computations. Some applications, e.g., applicationsin the Microsoft® Office suite offered by Microsoft Corporation(Redmond, Wash.), are visible to the user and enable user interactionthrough a user interface that is provided to the user when anapplication is launched. Other applications, e.g., virus scanningapplications, operate in the background and do not always require userinteraction. Applications that offer a user interface can include one ormore menus including selectable objects that represent operations that asoftware application can perform upon being selected by a user. Anoperation can be represented by a corresponding title containing text oricon containing an image, or both, and displayed in the user interface.

SUMMARY

In one example, a user can enter a query in a text box displayed in auser interface of an application, e.g., Microsoft® Office PowerPoint®,offered by Microsoft Corporation (Redmond, Wash.). The query can berelated to an operation that the application can perform, e.g.,inserting pictures from a scanner or camera into a slide. An objectcorresponding to the operation made available through the userinterface, e.g., as a choice in a menu of operations, can be identifiedto the user in response to receiving the query. The application'sresources, e.g., program files related to the application, can besearched and the location of the object corresponding to the operationcan be identified. Subsequently, the location of the object, e.g., themenu where the object resides, can be exposed, e.g., displayed in theuser interface, such that the object becomes apparent to the user. Inaddition, the searched operation can be performed automatically—that is,without receiving further input from the user.

In one aspect, a computer-implemented method includes searchingresources related to an application in response to receiving a queryfrom a user, identifying an operation in the application that satisfiesthe query, and exposing an object, within the application, that providesaccess to the identified operation.

This, and other aspects, can include one or more of the followingfeatures. Exposing the object can include causing a location, where theobject resides, to be displayed. The method can further includehighlighting the object. The location can include one or more of a menu,a tab, a dialog box, a sub-menu, a sub-tab, or a sub-dialog box. Themethod can further include identifying content that satisfies the query,wherein the content is unrelated to the operation, and presenting acontent location of the identified content. The method can furtherinclude receiving the query from the application, wherein the query ispresented in a user interface of the application.

In another aspect, a medium bearing instructions to enable one or moremachines to perform operations is described. The operations includesearching resources related to an application in response to receiving aquery from a user, identifying an operation in the application thatsatisfies the query, and exposing an object, within the application,that provides access to the identified operation.

This, and other aspects, can include one or more of the followingfeatures. Exposing the object can include causing a location, where theobject resides, to be displayed. The operations can further includehighlighting the object. The location can include one or more of a menu,a tab, a dialog box, a sub-menu, a sub-tab or a sub-dialog box. Theoperations can further include identifying content that satisfies thequery, wherein the content is unrelated to the operation, and presentinga content location of the identified content. The operations can furtherinclude receiving the query from the application, wherein the query ispresented in a user interface of the application.

In another aspect, a system comprising an engine is described. Theengine is configured to search resources related to an application inresponse to receiving a query from a user, identify an operation in theapplication that satisfies the query, and expose an object within theapplication, that provides access to the identified operation.

This, and other aspects, can include one or more of the followingfeatures. The engine can further be configured to cause a location,where the object resides, to be displayed. The engine can further beconfigured to highlight the object. The location can include one or moreof a menu, a tab, a dialog box, a sub-menu, a sub-tab, or a sub-dialogbox. The engine can further be configured to identify content thatsatisfies the query, wherein the content is unrelated to the operation,and present a content location of the identified content. The engine canfurther be configured to receive the query from the application, whereinthe query is presented in the user interface of the application.

In another aspect, a computer-implemented method is described. Themethod includes searching resources related to an application inresponse to receiving a query, identifying an operation in theapplication that satisfies the query, and automatically performing theoperation.

This, and other aspects, can include one or more of the followingfeatures. The method can further include automatically performing theidentified operation without further input from the user. The method canfurther include identifying content that satisfies the query, whereinthe content is unrelated to the operation, and presenting a location ofthe identified content. The method can further include receiving thequery from the application, wherein the query is presented in a userinterface of the application.

The systems and techniques described here may provide one or more of thefollowing advantages. Enabling a user to search for locations of objectsrepresenting operations that can be performed by an application canallow the user to exploit the features of an application by identifyingoperations that may not be apparent to the user. Automaticallyperforming an operation upon identifying an operation for which the useris searching can enhance user experience. Identifying operations that anapplication enables a user to perform in response to receiving a querycan help educate a user about the features of the application. Enablinga user to search for operations can allow the user to quickly find andexecute operations which may be difficult or non-obvious to otherwisefind. Further, users can find operations in the application withoutknowing or caring about the navigational structure of the application.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features andadvantages will be apparent from the description and drawings, and fromthe claims

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic of an example of a system for searching forresources related to an application.

FIG. 2 is a schematic of an example of a user interface of anapplication.

FIG. 3 is a schematic of an example of a user interface of anapplication.

FIG. 4 is a schematic of an example of a system for searching foroperations related to an application.

FIG. 5 is a flow chart of an example of a process for searching for anoperation related to an application.

FIG. 6 is a flow chart of an example of a process for searching for anoperation related to an application.

DETAILED DESCRIPTION

FIG. 1 depicts an example of a schematic of a system 100 for searchingfor resources related to an application. The system 100 can include anengine 105 configured to enable a user to interact with an applicationinstalled on the engine 105. The system 100 can include a display device110 operatively coupled to the engine 105, which can display a userinterface related to the installed application. The system 100 caninclude input devices, e.g., a keyboard 115, and a pointing device 120,e.g., a mouse, operatively coupled to the engine 105 which the user canuse to interact with the application, where the results of theinteraction can be displayed, e.g., in the user interface related to theapplication. In some implementations, the engine 105 can be operativelycoupled to a remote engine 125 over a network 130, where the remoteengine 125 can be, e.g., a host of a web site, e.g., www.google.com, andthe network 130 can be the Internet. In some implementations, the engine105, the display device 110, and the input devices 115 and 120, can bepart of a computer system, e.g., a desktop computer, a laptop computer,a personal digital assistant (PDA), and the like. The application can beany software application, e.g., Microsoft® Office PowerPoint®, that canbe installed on the engine 105. The application can include applicationresources, e.g., program files related to the application, which, wheninstalled on the engine 105, can be executed by the engine 105 toperform operations such as displaying a user interface on the displaydevice 110, receiving input from a user via the input devices, producingoutput that can be displayed in the user interface on the display device110, performing operations in the background that are not visible on thedisplay device 110, and the like. The engine 105 can be operativelycoupled to the remote engine 125 through the network 130 via wired orwireless interfaces.

An application can be installed on the engine 105, e.g., using a CD-ROMincluding the application or by downloading the application from theremote engine 125 over the network 130. Resources, e.g., program files,related to the application can be made available to the engine, e.g., bestored on the engine 105, such that the engine 105 can execute, e.g.,instructions included in the resources, to cause a user interface to bedisplayed on the display device 110 operatively coupled to the engine105, and an object, e.g., an icon, to be displayed on the display device110. In addition, the resources can include instructions to enable auser to perform operations using the application. A user can use aninput device, e.g., the keyboard 115 or the mouse 120 or both, to selectthe object representing the application. The engine 105 can detect thatthe user has selected the object and, in response, can launch theapplication, where, in some implementations, launching the applicationcan include the engine 105 displaying a user interface related to theapplication on the display device 110 and enabling the user to interactwith the user interface using the input devices, e.g., the keyboard 115,the mouse 120, and the like. In some implementations, the applicationcan allow the user to perform several operations, e.g., add content,modify content, delete content, and the like, the results of whichoperations can be viewed in the user interface.

The application user interface can include several objects where eachobject can correspond to one or more operations that the applicationenables the user to perform. For example, opening a file is an operationthat an application enables a user to perform, for which the applicationuser interface can include a corresponding object, e.g., a clickablebutton in a menu, which a user can select. When the user selects theobject, e.g., using the input devices, a default folder is displayed inthe user interface, which the user can use as a starting location toidentify the file that the user wishes to open. Similarly, the engine105 can receive an input from a user through the input devices, wherethe input includes selecting an object corresponding to an operation,displayed in a menu of operations in the user interface of theapplication. In response to the selection, the engine 105 can performthe operation corresponding to the selected object, based on theapplication resources, stored on the engine 105, that define theoperation to be performed.

In some implementations, the engine 105 can display a user interface,related to an application, on the display device 110, in which the usercan enter a query, where the query relates to an operation of theapplication. For example, the user interface can include a text box intowhich a user can enter a query, e.g., a string of text, where the querycan be related to an operation that can be performed by the application.The engine 105 can search the application resources, e.g., program filesrelated to the application, to identify the one or more operations thatsatisfy the query. Subsequently, the engine 105 can identify the objectcorresponding to the searched operation, e.g., the menu where the objectcorresponding to the searched operation resides, and can present theidentified object as a search result. For example, the engine 105 candisplay the menu containing the object corresponding to the searchedoperation and highlight the object. Alternatively, or in addition, theengine 105 can cause the searched operation to be performed.

FIG. 2 depicts a schematic of a user interface 205 of an applicationdisplayed in the display device 110 operatively coupled to the engine105. The engine 105 can display the user interface 205 in response toreceiving a selection from the user to launch the application to whichthe user interface 205 corresponds. The user interface 205 can includeseveral objects 210, 212, 214, 216, 218, 220, 222, and 224 which cancorrespond to operations that the application can enable a user toperform or an identifier for a collection of operations or both. Forexample, object 210 can be titled “File”, wherein object 210 representsa collection of operations, e.g., “Open”, “Close”, “Save”, “Import”, andthe like, where each operation under object 210 can be accessed byselecting object 210. In another example, object 224 can be “Convert toPDF” where selecting the object 224 can cause an operation to beperformed, wherein the operation, in this example, is conversion to PDF.

The user interface 205 can include a text box 230 into which a user canenter a query, e.g., a string of text by positioning the cursor 235,operated by the pointing device 120, in the text box 230, and enteringtext using the keyboard 115. The text can be a query related to anoperation that can be performed by the application. For example, one ofthe operations that can be performed by the application can be titled“3-1”. The object 3-1, corresponding to the operation 3-1, which theuser can select in order to perform the operation 3-1, may not beapparent to the user because the object 3-1 may reside in a menu underobject 214. The user can enter the title “3-1” or text that is relatedto the operation 3-1 in the text box 230 to search for the operation3-1. In response, the engine 105 can identify that the operation 3-1satisfies the query in the text box 230, identify that the operation 3-1is represented by the object 3-1 residing in the menu under object 214,and expose the object 3-1 to the user, e.g., display the menu underobject 214 in the user interface 205 as a search result 240. Inaddition, the engine 105 can highlight the object 3-1, e.g., by fillingthe object 3-1 with a color that can be readily discerned by the user.In this manner, the engine 105 can present the user with the location ofan operation in response to the user's query, and enable the user toselect the operation, e.g., by positioning the cursor 235 on the object3-1 and selecting the object.

In some implementations, the engine 105 can perform the operation 3-1upon identifying that the user was searching for the operation 3-1. Forexample, the application can be Microsoft® Office PowerPoint®, and theoperation can be inserting a picture from a camera or scanner. Theobject that the user needs to select to insert a picture from a cameraor scanner may reside in a menu on the user interface 205, and may,thus, not be apparent to the user. The user can enter “inserting apicture from a camera or scanner” as a query in the text box 230. Theengine 105 can parse the query and identify that the operation that theuser is looking for is inserting a picture from a camera or scanner.Further, by searching the application resources, the engine 105 canidentify that the menu option to enable a user to perform the requiredoperation resides under “Insert→Picture→From Scanner or Camera . . . ”.The engine 105 can cause the hierarchical menu, including the objecttitled “From Scanner or Camera . . . ” to be displayed on the userinterface 205, and, in addition, perform the operation. Thus, a responseto a search for “inserting a picture from a camera or scanner” can bethe engine 105 accessing a scanner or camera operatively coupled to theengine 105 and enabling the user to select a picture located on thescanner or camera by displaying a selection window in the user interface205. Although Microsoft® Office Powerpoint® and inserting a picture fromcamera or scanner as presented as representative examples, the aboveoperation can be performed for any application and for any feature in anapplication. In some implementations, the user can perform the operationwithout any further input from the user. In other implementations, theuser can identify an operation, present the identified operation to theuser, and confirm that the user intended to select the identifiedoperation. If the user confirms that the identified operation is theoperation that the user was searching, then the engine 105 can performthe identified operation. For example, upon identifying an operation,the engine 105 can display a window including the name of the operation,a question asking if the identified operation is the one that the userwas searching, and selectable buttons, such as “Yes” and “No” buttons.The engine 105 can perform the identified operation if the user selects“Yes.”

FIG. 3 depicts a schematic of a user interface 305 of an applicationdisplayed in the display device 110 operatively coupled to the engine105. The engine 105 can display the user interface 305 in response toreceiving a selection from the user to launch the application to whichthe user interface 305 corresponds. The user interface 305 can includeseveral objects 310, 312, 314, 316, 318, 320, 322, and 324 which cancorrespond to operations that the application can enable a user toperform or an identifier for a collection of operations or both. Theuser interface 305 can include a text box 330 in which a user can entera query related to an operation that the application enables a user toperform, content that can be found in, e.g., a data file which the usercreated with the application, a topic related to applicationcapabilities, and the like. When the user enters the query in the textbox 330, the engine 105 can search the application resources to identifyan operation that satisfies the query. In addition, the engine 105 cansearch data files, created using the application, for content thatsatisfies the query and, further, can retrieve content, related to theapplication that satisfies the query, residing on one or more remoteengines 125, connected to the engine 105 through the network 130. Thedata files can reside on one or more storage devices operatively coupledto the engine 105.

For example, the user can enter “3-1” as the query in the text box 330in the user interface 305, where the query can be a string of text. Thestring of text (e.g., “Insert”) can correspond to an operation that canbe performed by the application, e.g., inserting an object. The engine105 can search the application resources to identify the location of theobject containing the text “3-1” that corresponds to the operation thatsatisfies the query “3-1” entered in the text box. In addition, theengine 105 can traverse the contents stored in one or more storagedevices operatively coupled to the engine 105 to identify content, e.g.,data files created using the application, that satisfy the query, e.g.,contain the text “3-1.” Further, the engine 105 can interact with one ormore search engines, e.g., Google search engine, to search one or moreremote engines 125 connected to the engine 105 over the network 130 toidentify content that satisfies the query, e.g., web pages on web sitesthat describe the operation that satisfies the query “3-1.” The one ormore search engines can receive the query from the engine 105 over thenetwork, search network locations, e.g., web pages of web sites, forcontent relevant to the query, and can transfer the results of thesearch to the engine 105. The engine 105 can display the search results340 in the user interface 305 where the results can include the locationof the object “3-1” in the application, the names of one or more datafiles including content that satisfies the query, and identifiers, e.g.,uniform resource locators (URLs), pointing to the location of one ormore network locations that include content that satisfies the query. Insome implementations, the engine 105 can rank the results, where theranking can be based, e.g., on the location that was searched. Forexample, the engine 105 can present application operations that satisfythe query first, followed by content stored on one or more storagedevices operatively coupled to the engine 105, and by content obtainedfrom network locations. The results can further be ranked, e.g., basedon the relevance of each result with respect to the search query. Insome implementations, the universe of content that can be searched,including the application resources, content on storage devices, contentat remote locations, and the like, can be chosen by the user. Forexample, the user can instruct the engine 105 to limit the search toonly application resources. Alternatively, the user can instruct theengine to include content on storage devices and application resourceswhile searching.

FIG. 4 depicts a schematic of an example of a system configured toidentify locations of objects representing operations that anapplication can perform in response to receiving a query related to theoperation. In some implementations, the system can include a displaydevice 110 operatively coupled to the engine 105. The application can beinstalled on the engine 105, where installing the application caninclude making available to the engine 105 application resources 405,e.g., instructions to perform operations in the application, needed tooperate the application, where the application resources 405 can includetextual resources 410 and non-textual resources 415. For example, theapplication resources 405 can be made available to the engine 105 bystoring the textual resources 410 and non-textual resources 425 on theengine 105 when the application is installed and enabling the engine 105to access the application resources 420. The application resources 405can cause the display of the user interface 405 on the display device110 when the user launches the application. Further, when the userselects content displayed in the user interface 420 and an objectrepresenting an operation, e.g., an item titled “Cut” on a menu titled“Edit”, the application resources 425 can cause the engine 105 toperform the “Cut” operation on the content selected by the user. Thetextual resources 410 can include resources relating to objectsdisplaying strings of text in the user interface 420 to operations thatcan be performed by selecting such objects. The non-textual resources415 can include resources relating to images displayed in the userinterface 420 to operations that can be performed by selecting suchimages. Further, in some applications, an object displaying a string oftext and an image can correspond to the same operation. Suchrelationships between objects displaying strings of text, correspondingimages, and corresponding operations can also be stored in applicationresources 405. For example, the application resources 405 can be severaldata files hierarchically stored on a desktop file system, where suchdata files can include the user interface 420, provided by theapplication, that can be displayed on the display device 110 when theapplication is launched.

The engine 105 can launch the application in response to user input,e.g., a selection of an object, such as an icon, displayed on thedisplay device 110 using the keyboard 115 or the mouse 120. The engine105 can retrieve from application resources 405, the user interface 420,and expose the user interface 420, e.g., display the user interface 420on the display device 110. The engine 105 can enable a user to interactwith the user interface 420 using the input devices, and, in response touser inputs, perform operations where the operations performed aredetermined by the application resources 405. In some implementations,the user interface 420 can include a search facility 425 into which theuser can enter a query, e.g., a string of text. The engine 105 caninclude a query parsing module 430 that can receive the query entered inthe search facility 425 and can parse the query, e.g., separate thequery into one or more word phrases based on the query content.Subsequently, the engine 105 can search application resources 405 for anoperation that satisfies the query. For example, the query can be theword “Cut”, which the query parsing module 430 can identify and comparewith textual resources 410 in application resources 405 to identifyinstructions that need to be executed to perform the “Cut” operation.Further, the engine 105 can search application resources 405 to identifythe location, in the user interface, of an object representing the “Cut”operation.

In some implementations, the engine 105 can include an operationexecution module 435 configured to present the identified operation onthe user interface 420. For example, the engine 105 can identify thelocation of the text “Cut” in the application resources 405 and theinstructions to perform the “Cut” operation, and present the locationand the instructions to the operation execution module 435. Theoperation execution module 435 can display, in the user interface 420,the location of the object including the text “Cut”, which the user canselect to perform the cutting operation. In other implementations, inaddition to displaying the location of the object cut in the userinterface 420, the operation execution module 435 can perform theoperation that the user searched for.

FIG. 5 depicts a flow chart of an example of a process 500 foridentifying an operation related to an application in response toreceiving a query. The process 500 can receive a query in a userinterface at 505. For example, the engine 105 can display a userinterface in a display device 110, where the user interface can berelated to an application e.g., a software application such asMicrosoft® Office PowerPoint®, and can be made available to the engine105 when the application is installed in the engine 105. The userinterface can include a text box into which a user can enter a query,e.g., as a string of text.

The process 500 can search application resources to identify anoperation that satisfies the query at 510. For example, when anapplication is installed on the engine 105, application resources,needed to perform the operations of the application, can be stored on astorage device operatively coupled to the engine 105. The applicationresources can include the user interface and instructions to enable theengine 105 to display the user interface in a display device 110 inresponse to receiving instructions from the user. A user interface canbe displayed on the display device 110 into which a user can enter aquery, e.g., a string of text. The engine 105 can search the applicationresources to identify resources that satisfy the received query, e.g.,compare the text in the query with the application resources to identifytext in the application resources that match the text. In this manner,the engine 105 can identify operations that the application enables auser to perform, where the title of the operation satisfies the receivedquery. For example, in response to a query “remove content”, the process500 can search the application resources and identify the operation“Delete”.

The process 500 can expose an object that provides access to theidentified operation 515, e.g., on the user interface on the displaydevice 110. For example, the engine 105 can display the identifiedoperation that satisfied the query on the user interface. The identifiedoperation can be the title of an object that can reside in the userinterface, e.g., in a menu. In response to receiving the query, theengine 105 can display the menu, in which the object corresponding tothe identified operation resides, in the user interface.

FIG. 6 depicts a flow chart of an example of a process 600 forautomatically performing the operation that is identified upon searchingthe application resources. The process 600 can search applicationresources to identify an operation that satisfies the query at 610. Forexample, when an application is installed on the engine 105, applicationresources, needed to perform the operations of the application, can bestored on a storage device operatively coupled to the engine 105. Theapplication resources can include the user interface and instructions toenable the engine 105 to display the user interface in a display device110 in response to receiving instructions from the user. A userinterface can be displayed on the display device 110 into which a usercan enter a query, e.g., a string of text. The engine 105 can search theapplication resources to identify resources that satisfy the receivedquery, e.g., compare the text in the query with the applicationresources to identify text in the application resources that match thetext. In this manner, the engine 105 can identify operations that theapplication enables a user to perform, where the title of the operationsatisfies the received query. For example, in response to a query“remove content”, the process 600 can search the application resourcesand identify the operation “Delete”.

The process 600 can automatically perform the identified operation at615. For example, if the search query is “make letters bold”, and theengine 105 identifies “Bold” as the operation that satisfies the query,the engine 105 can replace selected text in the user interface with boldtext. In some implementations, the process 600 can automatically performthe identified operation without further input, e.g., from a user. Inother implementations, the process 600 can confirm that the identifiedoperations are the operations that the user was searching for, prior toautomatically performing the identified operation.

Implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Implementationsof the subject matter described in this specification can be implementedas one or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a computer-readable medium forexecution by, or to control the operation of, data processing apparatus.The computer-readable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them. The term “data processing apparatus” encompassesall apparatus, devices, and machines for processing data, including byway of example a programmable processor, a computer, or multipleprocessors or computers. The apparatus can include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, or acombination of one or more of them. A propagated signal is anartificially generated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal, that is generated to encodeinformation for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub-programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto-optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio player, a Global Positioning System (GPS)receiver, to name just a few. Computer-readable media suitable forstoring computer program instructions and data include all forms ofnon-volatile memory, media and memory devices, including by way ofexample semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech,near-touch input, or tactile input.

Implementations of the subject matter described in this specificationcan be implemented in a computing system that includes a back-endcomponent, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front-endcomponent, e.g., a client computer having a graphical user interface ora Web browser through which a user can interact with an implementationof the subject matter described is this specification, or anycombination of one or more such back-end, middleware, or front-endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, e.g., a communicationnetwork. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and sewerarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the disclosure or of what maybe claimed, but rather as descriptions of features specific toparticular implementations of the disclosure. Certain features that aredescribed in this specification in the context of separateimplementations can also be implemented in combination in a singleimplementation. Conversely, various features that are described in thecontext of a single implementation can also be implemented in multipleimplementations separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. For example, in someimplementations, the user interface into which a user can enter thequery can be unrelated to any application and can be designed to enablea user to search for operations in several applications. When the userinterface receives a query, the engine 105 can search applicationresources related to one or more applications installed on the engine105 and identify operations in one or more of the several applicationsthat satisfy the search query. For example, if the user enters “Crop” asa search query in the user interface, the engine 105 can searchapplication resources related to several applications and identify allapplications that enable a user to perform the “Crop” operation.Subsequently, the engine 105 can generate identifiers to each operationcorresponding to each application and provide the generated identifierto the user, wherein, when the user selects an identifier, the userinterface of the corresponding application and the location of theobject representing the operation that matched the user's query can bedisplayed in the display device 110.

In some implementations, in addition to displaying the location ofobjects representing the operations that satisfy the query, the engine105 can display icons representing the operations. For example, the“Open” operation, that enables a user to open a file in an applicationcan also be represented, by an icon, e.g., a half-open manila folder. Ifthe operation that satisfies the query presented in the user interfaceis the “Open” operation, in addition to displaying the location of theobject corresponding to the “Open” operation in the user interface, theengine 105 can also display the icon representing the “Open” operationon the user interface. If the icon is already displayed in the userinterface, the engine 105 can highlight the icon so that it is readilydiscerned by the user. The engine 105 can identify the iconcorresponding to an operation by scanning the application resourcesstored in the engine 105 when the application was installed.

In some implementations, the engine 105 can index the applicationresources to search for operations that satisfy queries quicker and in amore efficient manner. The engine 105 can include a crawler configuredto traverse the application resources, e.g., the data files that aremade available to the engine 105 when the application is installed, tocreate the indexed application resources.

In some implementations, the resources of the engine 105 can be includedwith the application such that when the application is installed, theresources of the engine 105 are automatically installed or are providedto a user as an option to install. In other implementations, the engine105 and the resources of the engine 105 can be made available as aseparate feature which can be added to any application, e.g., in theform of a plug-in.

In some implementations, the application resources can reside at aremote location, e.g., a host of a web site, to which the engine 105 canbe operatively coupled over the network. In such implementations, when aquery searching for the location of objects representing operations isentered in a user interface, the engine 105 can search the applicationresources residing at the remote location to identify locations ofobjects representing the operations that satisfy the query. Theidentified locations can be displayed on a display device 110operatively coupled to the engine 105.

In some implementations, the universe of resources that is searched inresponse to receiving a search query can be controlled by a user. Forexample, the user can choose to search only application resources inresponse to receiving a search query. In other implementations, the usercan choose to include content stored in one or more storage devicesoperatively coupled to the engine 105 and the content available atremote locations operatively coupled to the engine 105 via the network,in addition to the application resources, in the universe of resourcesthat are searched. In some implementations, the objects representing theoperations can reside in one or more of tabbed windows, dialog boxes,sub-menus, sub-tabbed windows, sub-tabbed dialog boxes, and the like.Accordingly, other implementations are within the scope of the followingclaims.

1. A method comprising: executing a software application on a computingdevice, the software application having a user interface, the userinterface including a menu that provides access to operations that thesoftware application is configured to perform; receiving a usersubmitted query; in response to receiving the user submitted query,searching software application resources to identify an operation thatsatisfies the user submitted query, wherein the identified operation isone of the operations that the software application is configured toperform, and wherein the identified operation corresponds to a menu itemlocated within the menu of the user interface wherein activation of themenu item causes the identified operation to be performed; and inresponse to identifying the operation, displaying the menu itemcorresponding to the identified operation in the user interface by atleast visually expanding at least a portion of the menu to display themenu item.
 2. The method of claim 1, further comprising: generatingquery results that include the identified operation that satisfies thequery.
 3. The method of claim 2, further comprising: displaying, in aresult list of the user interface of the software application, the queryresults.
 4. The method of claim 3, further comprising: receiving aselection of the identified operation using the result list; andperforming the selected operation.
 5. The method of claim 3, whereindisplaying the query results in the result list comprises: sorting,based at least in part on relevance of the query results to the query,the generated query results into a sorted order; and displaying thequery results in the sorted order.
 6. The method of claim 1, furthercomprising: identifying content in addition to the operation thatsatisfies the query; and displaying an indication of the identifiedcontent.
 7. The method of claim 1, wherein displaying the menu item inthe user interface comprises highlighting the menu item.
 8. The methodof claim 1, wherein displaying the menu item in the user interfacecomprises displaying, in the user interface, a navigational structure ofthe menu that is used to navigate to the location of the menu itemwithin the menu.
 9. The method of claim 1, wherein the menu comprises ahierarchical menu that includes two or more hierarchical levels of menuitems, and wherein displaying the menu item in the user interfacecomprises visually expanding at least a portion of the menu to thehierarchical level at which the menu item is located.
 10. The method ofclaim 1, wherein the user interface comprises a display device fordisplaying the menu and one or more input devices for receiving inputfor the query.
 11. The method of claim 10, wherein the one or more inputdevices includes a voice input device, and wherein the query comprises avoice-based query that is received via the voice input device.
 12. Anon-transitory computer-readable storage medium comprising instructionsthat, when executed, cause one or more processors to: execute a softwareapplication on a computing device, the software application having auser interface, the user interface including a menu that provides accessto operations that the software application is configured to perform;receive a user submitted query; in response to receiving the usersubmitted query, search software application resources to identify anoperation that satisfies the user submitted query, wherein theidentified operation is one of the operations that the softwareapplication is configured to perform, and wherein the identifiedoperation corresponds to a menu item located within the menu of the userinterface wherein activation of the menu item causes the identifiedoperation to be performed; and in response to identifying the operation,display the menu item corresponding to the identified operation in theuser interface by at least visually expanding at least a portion of themenu to display the menu item.
 13. The computer-readable storage mediumof claim 12, further comprising instructions that, when executed, causeone or more processors to: generate query results that include theidentified operation that satisfies the query.
 14. The computer-readablestorage medium of claim 13, further comprising instructions that, whenexecuted, cause one or more processors to: display, in a result list ofthe user interface of the software application, the query results. 15.The computer-readable storage medium of claim 14, further comprisinginstructions that, when executed, cause one or more processors to:receive a selection of the identified operation using the result list;and perform the selected operation.
 16. The computer-readable storagemedium of claim 14, wherein the instructions to display the queryresults in the results list comprise instructions to: sort, based atleast in part on relevance of the query results to the query, thegenerated query results into a sorted order; and display the queryresults in the sorted order.
 17. The computer-readable storage medium ofclaim 12, wherein the user interface comprises a display device fordisplaying the menu and one or more input devices for receiving inputfor the query.
 18. The computer-readable storage medium of claim 17,wherein the one or more input devices includes a voice input device, andwherein the query is a voice-based query that is received via the voiceinput device.
 19. The computer-readable storage medium of claim 12,wherein the instructions to display the menu item in the user interfacecomprise instructions to: highlight the menu item.
 20. Thecomputer-readable storage medium of claim 12, wherein the instructionsto display the menu item in the user interface comprise instructions to:display, in the user interface, a navigational structure of the menuthat is used to navigate to the location within the menu of the menuitem.
 21. The computer-readable storage medium of claim 12, wherein themenu is a hierarchical menu that includes two or more hierarchicallevels of menu items, and wherein the instructions to display the menuitem in the user interface comprise instructions to: visually expand atleast a portion of the menu to the hierarchical level at which the menuitem is located.
 22. A computing device comprising: a display; one ormore processors; and one or more applications operable by the processorto present, on the display, a user interface having a menu that providesaccess to operations that the software application is configured toperform, and wherein the software application is operable by theprocessor to: receive a user submitted query, in response to receivingthe user submitted query, searching software application resources toidentify an operation that satisfies the query, wherein the identifiedoperation is one of the operations that the software application isconfigured to perform, and wherein the identified operation correspondsto a menu item located within the menu of the user interface whereinactivation of the menu item causes the identified operation to beperformed, and in response to identifying the operation display the menuitem corresponding to the identified operation in the user interface byat least visually expanding at least a portion of the menu to displaythe menu item.
 23. The computing device of claim 22, wherein theapplication is further operable by the processor to generate queryresults that include the identified operation that satisfies the query.24. The computing device of claim 23, wherein the application is furtheroperable by the processor to present, in a result list of the userinterface, the query results.
 25. The computing device of claim 24,wherein the application is further operable by the processor to receivea selection of an operation of the at least one operation that isresponsive to the query using the result list, and perform the selectedoperation.
 26. The computing device of claim 24, wherein the applicationis operable by the processor to present the query results in the resultlist by at least being operable to: sort, based at least in part onrelevance of the query results to the query, the generated query resultsinto a sorted order; and display the query results in the sorted order.27. The computing device of claim 22, wherein the application isoperable by the processor to display the menu item in the user interfaceby at least being operable to: highlight the menu item on the displaydevice.
 28. The computing device of claim 22, wherein the application isoperable by the processor to display the menu item in the user interfaceby at least being operable to: display, on the display device, anavigational structure of the menu that is used to navigate to thelocation within the menu of the menu item.
 29. The computing device ofclaim 22, wherein the menu is a hierarchical menu that includes two ormore hierarchical levels of menu items, and wherein the application isoperable by the processor to display the menu item in the user interfaceby at least being operable to: visually expand at least a portion of themenu on the display device to the hierarchical level at which the menuitem is located.